Methods, systems and computer readable media for diagnosing network function virtualization performance

ABSTRACT

Performance issues in a service function chain having a plurality of resources and a plurality of network functions each having a network function queue are diagnosed. Each network function queue is monitored and queueing information for input packets for each of the plurality of network functions is dumped to a data store. Each resource that is under contention is identified as well as which of the network functions is a contender for the resources. A diagnosing algorithm is used to diagnose performance problems and an impact graph for each victim packet is generated. A summary of results as a list of rules is then provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/714,117, filed Dec. 13, 2019. The contents of each of the foregoingare hereby incorporated by reference into this application as if setforth herein in full.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under grant 1413978awarded by the National Science Foundation. The government has certainrights in the invention.

TECHNICAL FIELD

The present disclosure relates to network communication methods systemsand computer readable media. More particularly, the disclosure relatesto a method, system, and computer program for diagnosing performance ofnetwork function virtualization.

BACKGROUND

A telecommunication's network operator can deploy a large number ofservices in its networks. These services include but are not limited tovoice, data and video services. In modern networks, NFV (networkfunction virtualization) has been replacing middleboxes, because NFVruns network functions (NFs) on software platform, which achieves highflexibility in developing new functionalities, helps network operatorsto deploy network functions easily, and requires low cost in networkfunction management. However, compared to middleboxes, NFV is moreerror-prone, and traffic in NFV often experiences long tail latency oroccasional packet drops. Usually, people simply blame the “software”nature and neglect it. However, as the long tail latency becomes moresignificant in applications' performance, there is a need to betterunderstand the NFV performance issue.

Service function chains consisting of NFs and virtual network functions(VNFs) are the easiest way to realize these services. Each servicefunction chain can have many NFs/VNFs and each network function can havemany instances. With a large number of network function instances thatmight share same physical infrastructure it is important that a networkoperator has the ability to quickly detect latency and packet dropissues. As these issues can amplify into service issues impactingquality of service/experience of the end users. With a large set ofnetwork function instances, it is difficult for network operators tomanually analyze the traffic logs to understand the service problems.Additionally, these problems can disappear even before network operatoris done with analysis.

Performance issues in NFV stem from the “temporal resource contention”.There are various types of resource contentions that can impact NFVperformance, including the queue, the CPU, the cache, etc. Each resourcehas different types of contenders:

Queue: bursts of flows arriving into the queue.

CPU: bursts of flows at other NFs, interrupts, or other CPU usage.

Cache: bursts of flows at other NFs, or other cache usage.

Diagnosis in NFV is challenging. First, usually there are multipleresource contentions mixed together, and each could involve manycontenders. Each NF may be bound with a queue, cache resource, and CPUresource. It is hard to figure out how each resource contentioncontributes to the problem. Second, a local view of diagnosis is notsufficient. So, for example, if the performance problem happens in afirewall, then checking resource contentions within firewall (queue infirewall, cache and CPU) is not enough, because the root cause can alsocome from an NAT (e.g., the NAT sends out bursts of traffic), which is ahop away from the firewall. Third, the long latency of a packet is notonly impacted by the resource contention during the processing of it,but also the contention in history.

There is a need to monitor NFV performance and diagnose performanceproblems without the need to manually analyze traffic logs to understandthe service problems.

SUMMARY

One general aspect includes a method for diagnosing performance issuesin a service function chain having a plurality of resources and aplurality of network functions each having a network function queue. Themethod includes monitoring each network function queue and dumpingqueueing information for input packets for each of the plurality ofnetwork functions. The method further includes identifying each of theplurality of resources that is under contention and identifying which ofthe plurality of network functions is a contender for the each of theplurality of resources. A diagnosing algorithm is used to diagnoseperformance problems and an impact graph for each victim packet isgenerated. A summary of results as a list of rules is then provided.

Implementations may include one or more of the following features. Themethod where the performance issues include long tail latency or packetdrops. The method where the step of using a diagnosing algorithmincludes providing an impact score for each of the plurality ofresources that is under contention. The method where the step ofidentifying each of the plurality of resources that is under contentionincludes using resource counters to determine how each of the pluralityof resources is under contention. The method where the resource countersinclude symptom counters, resource measurement counters and trafficmeasurement counters. The method where the step of using a diagnosingalgorithm includes running a score back-propagation step for determiningwhich of the plurality of network functions propagate an impact to avictim package. The method further including running the scoreback-propagation step recursively until all root nodes areback-propagated.

One general aspect includes a system for diagnosing performance issuesin a service function chain including a plurality of resources; and aplurality of network functions each having a network function queue. Thesystem includes a processor and a non-volatile computer memory forstoring computer instruction coupled to the processor, where processor,responsive to executing the computer instructions, performs thefollowing operations: monitoring each network function queue; dumpingqueueing information for input packets for each of the plurality ofnetwork functions; identifying each of the plurality of resources thatis under contention; identifying which of the plurality of networkfunctions is a contender for the each of the plurality of resources;using a diagnosing algorithm to diagnose performance problems;generating an impact graph for each victim packet and generating asummary of results as a list of rules.

Implementations may include one or more of the following features. Thesystem where the performance issues include long tail latency or packetdrops. The system where using a diagnosing algorithm includes providingan impact score for each of the plurality of resources that is undercontention. The system where identifying each of the plurality ofresources that is under contention includes using resource counters todetermine how each of the plurality of resources is under contention.The system where the resource counters include symptom counters,resource measurement counters and traffic measurement counters. Thesystem where using a diagnosing algorithm includes running a scoreback-propagation step for determining which of the plurality of networkfunctions propagate an impact to a victim package. The system where theoperations further include running the score back-propagation steprecursively until all root nodes are back-propagated.

One general aspect includes a non-transitory, tangible computer-readablemedium having computer-executable instructions stored thereon which,when executed by a computer, cause the computer to perform a method fordiagnosing performance issues in a service function chain. The servicefunction chain having a plurality of resources and a plurality ofnetwork functions each having a network function queue. The methodperformed by the computer includes monitoring each network functionqueue; dumping queueing information for input packets for each of theplurality of network functions. The method further includes identifyingeach of the plurality of resources that is under contention andidentifying which of the plurality of network functions is a contenderfor the each of the plurality of resources. The method uses a diagnosingalgorithm to diagnose performance problems and generates an impact graphfor each victim packet. The method also includes generating a summary ofresults as a list of rules.

Implementations may include one or more of the following features. Thenon-transitory, tangible computer-readable medium where the performanceissues include long tail latency or packet drops. The non-transitory,tangible computer-readable medium where using a diagnosing algorithmincludes providing an impact score for each of the plurality ofresources that is under contention. The non-transitory, tangiblecomputer-readable medium where identifying each of the plurality ofresources that is under contention includes using resource counters todetermine how each of the plurality of resources is under contention.The non-transitory, tangible computer-readable medium where the resourcecounters include symptom counters, resource measurement counters andtraffic measurement counters. The non-transitory, tangiblecomputer-readable medium where using a diagnosing algorithm includesrunning a score back-propagation step for determining which of theplurality of network functions propagate an impact to a victim package.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a service chain to which the method fordiagnosing network function virtualization performance may be applied.

FIG. 2 is a block diagram illustrating the system design of a system fordiagnosing network function virtualization performance.

FIG. 3 is a block diagram showing how software is instrumented in anactual server to collect counters and queue information.

FIG. 4 is a flowchart of a method for diagnosing network functionvirtualization performance.

FIG. 5 is a flowchart of a method implemented by an algorithm fordiagnosing network function virtualization performance.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS Glossary

AutoFocus is a traffic analysis and visualization tool that describesthe traffic mix of a link through textual reports and time series plots.Using traffic clusters, one can divide the traffic into meaningfulcategories. AutoFocus produces time series plots of the traffic mix witheach category in a different color. AutoFocus produces reports and plotsfor various time periods ranging from weeks to half hour intervals.Also, the user can drill down into separate pages for each category. Thefilter from the user interface allows drill down into arbitrarydirections. AutoFocus accepts two types of input: packet header tracesand NetFlow data. Both types of input can be sampled, but AutoFocus onlycompensates for the sampling in the reports that measure the traffic inbytes and packets, and not in those measuring the traffic in flows

CAT. Cache Allocation Technology (CAT) provides software control ofwhere data is allocated into the last-level cache (LLC), enablingisolation and prioritization of key applications.

CMT. Cache Monitoring Technology (CMT) allows an Operating System (OS)or Hypervisor/virtual machine monitor (VMM) to determine the usage ofcache by applications running on the platform.

Contenders. Contenders are network functions that contend for resources.

DPDK. DPDK is a Data Plane Development Kit that consists of libraries toaccelerate packet processing workloads running on a wide variety of CPUarchitectures. The main goal of the DPDK is to provide a simple,complete framework for fast packet processing in data planeapplications. The DPDK framework creates a set of libraries for specifichardware/software environments through the creation of an EnvironmentAbstraction Layer (EAL). The EAL hides the environment specifics andprovides a standard programming interface to libraries, availablehardware accelerators and other hardware and operating system (Linux,FreeBSD) elements. Once the EAL is created for a specific environment,developers link to the library to create their applications. Forinstance, EAL provides the frameworks to support Linux, FreeBSD, IntelIA-32 or 64-bit, IBM POWER9 and ARM 32- or 64-bit.

Firewall. A firewall is software used to maintain the security of aprivate network. Firewalls block unauthorized access to or from privatenetworks and are often employed to prevent unauthorized Web users orillicit software from gaining access to private networks connected tothe Internet. A firewall may be implemented using hardware, software, ora combination of both.

Five-tuple. A 5-tuple refers to a set of five different values thatcomprise a Transmission Control Protocol/Internet Protocol (TCP/IP)connection. It includes a source IP address/port number, destination IPaddress/port number and the protocol in use.

Impact Score. An Impact Score provides an indication of the impact that. . . .

Light-weight information. Lightweight information is information that isrelatively simpler than other information.

Local Score Allocation. Local score allocation is the allocation of animpact score based on the number of packets buffered in the queue due toresource contentions. The goal of local score allocation is to determinewhere the long latency of the packet comes from, i.e., how the longqueue is built up.

NAT. Network address translation (NAT) is a router function that enablespublic and private network connections and allows single IP addresscommunication. While there are many public networks worldwide, there isa limited number of private networks. NAT was introduced as aneffective, timely solution to heavy network volume traffic.

NF. Network Function (NF)—a functional building block within a networkinfrastructure, which has well-defined external interfaces and awell-defined functional behavior. In practical terms, a Network Functionis today often a network node or physical appliance.

NFV Chain. NFV chain defines a list of individual network functions andthe sequence policy and Service Level Agreement (SLA) requirements amongthese functions.

NFV. Network functions virtualization (NFV) is an initiative tovirtualize network services traditionally run on proprietary, dedicatedhardware. With NFV, functions like routing, load balancing and firewallsare packaged as virtual machines (VMs) on commodity hardware.

Performance Counter. Performance Counters are a form of performancemonitoring and debugging tool to aid performance testing ofapplications. These count a number of actions that both the applicationand the operating system perform.

Queue Length. Queue length is the number of processing jobs awaitingservice at the processor station; the waiting jobs include the jobswaiting to be processed as well as the jobs being considered for remoteexecution by the load balancing mechanism.

Resource Contention. In computer science, resource contention is aconflict over access to a shared resource such as random access memory,disk storage, cache memory, internal buses or external network devices.A resource experiencing ongoing contention can be described asoversubscribed.

Resource counters (CPU, process, memory, disk and IP network interfaceutilization counters) are components that count resource consumption ofthe respective resource caused by data packets belonging to a specificservice.

Resources are resources used by an NF node (e.g. CPU resources, cacheresources).

Victim Packet. Victim packet is a packet that is dropped because it goesabove a queue time threshold.

VNF. Virtual network functions (VNFs) are virtualized tasks formerlycarried out by proprietary, dedicated hardware. VNFs move individualnetwork functions out of dedicated hardware devices into software thatruns on commodity hardware. These tasks, used by both network serviceproviders and businesses, include firewalls, domain name system (DNS),caching or network address translation (NAT) and can run as virtualmachines (VMs).

VNF Chain. A network function involving multiple VNFs.

FIG. 1 is a block diagram illustrating a service chain 100 on which themethod for diagnosing network function virtualization may be applied.Service chain 100 may include an NAT 101 and may access resources suchas cache 103 and CPU 105. NAT 101 may be accessed by other root nodessuch as root node 107, and input traffic node 109. CPU 105 may beaccessed by other root nodes such as other usage node 111. NAT 101 mayaccess firewall 113 which in turn may access resources such as cash 103and CPU 115. CPU 115 may be accessed by root node 117 and the firewallmay be accessed by root node 119. Service chain 100 may also include anIDS 121 which may access CPU resource 123 in cache resource 125. IDS 121may be accessed by root node 127 and input traffic node 128. CPU 123 maybe accessed by root node 129 and Cache 125 may be accessed by root node131.

As shown in FIG. 1 the NAT 101 and the firewall 113 share a cache line.If there is a burst of traffic arriving to the NAT 101, which causesresource contention in the shared cache line then the result may beperformance problems in the firewall. The operator can choose to isolatethe cash line for NAT 101 and firewall 113, or alternatively choose torate limit the input traffic arriving to NAT 101.

Diagnosis in NFV is challenging. First, usually there are multipleresource contentions mixed together and each could involve manycontenders. As shown in FIG. 1, each NF node (e.g. NAT 101, firewall113, IDS 121) is bound with a queue, a cache resource, and CPU resource.It is difficult to figure out how each resource contention contributesto the problem. Second, a local view of diagnosis is not sufficient. Asshown in FIG. 1, the performance problem happens in the firewall 113,then checking resource contentions within the firewall 113 (i.e. thequeue in the firewall, cache in the CPU) is not enough, because the rootcause can also come from NAT 101 (e.g. NAT 101 sends out bursts oftraffic) which is a half away from the firewall 113. Third, the longlatency of a packet is not only impacted by the resource contentionduring the processing of it, but also the contention in history. Asshown in FIG. 1 NAT 101 has a long buffered queue due to CPU contention,and it sends out all buffer packets in a burst after the contentionresulting in problems in firewall 113. These problems can be overcome byan NFV performance diagnosis system in accordance with the presentdisclosure.

Illustrated in FIG. 2 is an NFV performance diagnosis system 200. TheNFV performance diagnosis system 200 includes a plurality of servers(e.g. server A 201 and server B 203 disposed in a network 204. Server A201 includes a plurality of network functions (e.g. NF1A 205, NF2A 207and NF3A 209), and server B 203 includes a plurality of networkfunctions (e.g. NF1B 211, NF2B 213 and NF3B 215). Associated with serverA is dumper A 217 and associated with server B is dumper B 219. Dumper A217 and dumper B 219 are components that accomplish the dumping of thequeuing information for each input packet for each NF. The packet levelqueuing information is transferred to a storage device such as disk 221.The packet level queuing information is the processed by a detaileddiagnosis application 223. The detailed diagnosis application 223generates an impact graph 225 which is the summarized at summarycomponent 227 and then converted into a list of rules for operators thatinclude “<when>, <which flow>, at <which hop>, suffers from problems,caused by <which resource contention>”.

FIG. 3 shows how software is instrumented in an actual server to collectcounters and queue information. Server 301 includes a plurality ofnetwork functions (e.g. NF1 303, NF2 305 and NF3 307). The server alsoincludes a module that monitors and allocates cache usage (e.g. IntelCMT/CAT 309) that acts as a performance counter and dumper. Server 301also may include a DPDK 311 which acts as a high speed processinglibrary and include queuing information dumper 313. An interface card323 may be provided to interface virtual functions VF1 315, VF2 317, VF3319 and VF4 321. The interface card 323 may include additional virtualfunctions.

Illustrated in FIG. 4 is a method for diagnosing network functionvirtualization 400.

In step 401, the method 400 monitors each NF queue. With the queuinginformation from all NFs, the experience of each packet, and how thetraffic pattern changes across different NFs can be determined. Themethod 400 can perform offline diagnosis to identify relevant resourcecontentions, contenders, and how contentions propagate their impact tothe performance problem. The key idea of method 400 is to leveragepacket-level queuing information. Method 400 will dump the queuinginformation for each input packet for each NF, and the informationincludes: 1) five-tuple, 2) timestamp, 3) queue length, and 4) packet ID(which is used for identifying each packet). The packet-level queuinginformation is significant because when performance problems happen, themethod 400 can determine how the input queue is built up, and whichpackets are responsible for the built-up queue. Second, the queuinginformation describes how the traffic pattern changes across differentNFs, and the traffic pattern change indicates the propagation of theimpact of resource contentions. Third, with five-tuples of each packet,the method 400 can generate traffic patterns that are significant inperformance problems.

In step 403, the method 400 dumps the queuing information to a datastore.

In step 405, the method 400 identifies the resources under contention.This is accomplished through the use of resource counters. For example,Symptom Counters may include determination of the End-to-end latency ofeach packet; the latency of each packet within each NF; the packet losswithin each NF; the packet loss rate for each flow; and the tail latencyfor each flow. Resource Measurement Counters may determine CPU usage foreach NF; Cache misses for each NF; Kernel lock usage for each NF; andMemory bandwidth for each NF. Traffic Measurement Counters may determinethe number of packets arrived at each NF; the number of packetsprocessed by each NF; the flow distribution received by each NF; and theflow distribution sent by each NF. With the resource counters, how eachresource is under contention at any time may be determined. As a result,for each performance problem, all of the resource contentions thatcontribute to the problem, as well as all relative contenders may bedetermined. Furthermore, we can also know how the resource contentionchanges the traffic pattern, and propagates its impact to the finalproblem.

In step 407, the method 400 identifies the contentions VNFs.

In step 409, the method 400 diagnoses the performance problems. Thediagnosis is performed by a diagnosis algorithm. As mentioned above, themethod 400 leverages packet-level queuing information to diagnose theperformance problem. Therefore, the method 400 will perform diagnosis onevery packet suffering from performance problems (either long latency orpacket drop). The diagnosis algorithm can be divided into 2 steps: 1)local score allocation and 2) score back-propagation.

The goal of the local resource allocation is to determine where the longlatency of the packet comes from, i.e., how the long queue is built up.This is accomplished by considering the “queuing interval” rather thanthe actual queue content. Queuing interval means the time period fromthe time when the queue started to build up to the time when a victimpacket arrives. Queuing interval is used because the impact of resourcecontentions can come from traffic in history. Another concept in thisstep is to define an impact score that can be comparable acrossdifferent resource contentions. The score is based on the number ofpackets buffered in the queue due to resource contentions. For example,either a burst of flows or CPU shortage can buffer packets in the queue,because the NF cannot process the high input load in time. In this way,the number of packets buffered in the queue represents how differenttypes of resource contentions impact the performance Therefore, ageneral way to quantify the impact of different types of contentions isprovided.

The goal of Score back-propagation is to determine which componentspropagate their impact to the victim packet. The key idea is to figureout the correct set of packets that help to propagate the impact. Forexample, if the impact is propagated by sending higher rate traffic, theset of packets in the higher rate traffic must be determined. If theimpact is propagated through a shared resource, then there is a need todetermine which NF is the contender, and which set of packets make theNF over-utilizes the resource. The score back-propagation will be runrecursively, until all scores are back propagated to the root nodes.After finishing, an impact graph for a single victim packet can beobtained.

In step 411 the method 400 generates an impact graph. The impact graphto show the diagnosis result. The impact graph consists of differentcomponents in the NFV chain, and the links between components describehow different components impact each other through different ways. FIG.1 shows an example of the impact graph. Each node is provided with ascore to show how resource contentions impact this node. Each link isprovided with a score to show how much impact is propagated. Forexample, the NAT 101 receives impact from input traffic 109 for 200scores, and receives 100 scores from CPU 105. It then propagates 100scores to the cache 103, and propagates 80 scores to the Firewall 113.The rest 120 scores represent how resource contentions impact the NAT101.

In step 413, the method 400 generates a list of rules. After getting theimpact graph for each victim packet, a list of rules that can summarizethe diagnosis result can be generated. The key idea is to find the rulebased on all independent impact graph. For each NF, an AutoFocusalgorithm may be used to find out the significant flow pattern. Based onpackets in the flow pattern and their impact graph, how all resourcecontentions propagate their impact to the flow pattern may be derived.An accurate rule for those victim packets may then be generated. Asstated above, the list of rules for operators, may include “<when>,<which flow>, at <which hop>, suffers from problems, caused by <whichresource contention>”. A score for each rule to quantify how theresource contention contributes to the problem may be provided.

FIG. 5 is a flowchart of a method 500 implemented by the diagnosisalgorithm.

In step 501 the diagnosis algorithm provides impact score for resourcesunder contention.

In step 503 the diagnosis algorithm runs a score back propagationrecursively.

In view of the many possible embodiments to which the principles of thisinvention may be applied, it will be recognized that the embodimentdescribed herein with respect to the drawing figures is meant to beillustrative only and should not be taken as limiting the scope ofinvention. For example, those of skill in the art will recognize thatthe elements of the illustrated embodiment shown in software may beimplemented in hardware and vice versa or that the illustratedembodiment can be modified in arrangement and detail without departingfrom the spirit of the invention. Therefore, the invention as describedherein contemplates all such embodiments as may come within the scope ofthe following claims and equivalents thereof.

What is claimed:
 1. A method, comprising: monitoring, by a processingsystem including a processor, network function queues of a plurality ofvirtual network functions that are bound to common physical resources;dumping, by the processing system, queueing information for inputpackets for each of the plurality of virtual network functions;identifying, by the processing system, from the queueing information,each of the common physical resources that is under contention;determining, by the processing system, a plurality of network functionsin a service chain that are contenders for the each of the commonphysical resources that is under contention; using, by the processingsystem, a diagnosing algorithm to diagnose performance problems based onthe determining step; generating, by the processing system, an impactgraph for each victim packet identified by the diagnosing algorithm as aresult of a performance problem; and generating, by the processingsystem, a summary of results of the diagnosing algorithm as a list ofrules.
 2. The method of claim 1, wherein the performance problemcomprises long tail latency or packet drops.
 3. The method of claim 1,wherein the using the diagnosing algorithm comprises providing an impactscore for each of the common physical resources that is undercontention.
 4. The method of claim 1, wherein the identifying each ofthe common physical resources that is under contention comprises usingresource counters to determine how each of the common physical resourcesis under contention.
 5. The method of claim 4, wherein the resourcecounters comprise symptom counters, resource measurement counters andtraffic measurement counters.
 6. The method of claim 1, wherein thediagnosing algorithm comprises running a score back-propagation step fordetermining which of the plurality of network functions propagate animpact to a victim packet.
 7. The method of claim 6, further comprisingrunning the score back-propagation step recursively until all root nodesare back-propagated.
 8. A device comprising: a processing systemincluding a processor; and a memory that stores executable instructionsthat, when executed by the processing system, facilitate performance ofoperations, the operations comprising: monitoring network functionqueues of a plurality of virtual network functions that are bound tocommon physical resources; dumping queueing information for inputpackets for each of the plurality of virtual network functions;identifying from the queueing information, each of the common physicalresources that is under contention; determining a plurality of networkfunctions in a service chain that are contenders for the each of thecommon physical resources that is under contention; using a diagnosingalgorithm to diagnose performance problems based on the determiningstep; generating an impact graph for each victim packet identified bythe diagnosing algorithm as a result of a performance problem; andgenerating a summary of results of the diagnosing algorithm as a list ofrules.
 9. The device of claim 8, wherein the performance problemcomprises long tail latency or packet drops.
 10. The device of claim 8wherein using the diagnosing algorithm comprises providing an impactscore for each of the common physical resources that is undercontention.
 11. The device of claim 8, wherein identifying each of thecommon physical resources that is under contention comprises usingresource counters to determine how each of the common physical resourcesis under contention.
 12. The device of claim 11, wherein the resourcecounters comprise symptom counters, resource measurement counters andtraffic measurement counters.
 13. The device of claim 8, wherein usingthe diagnosing algorithm comprises running a score back-propagation stepfor determining which of the plurality of network functions propagate animpact to a victim packet.
 14. The device of claim 13, wherein theoperations further comprise running the score back-propagation steprecursively until all root nodes are back-propagated.
 15. Anon-transitory, machine-readable medium, comprising executableinstructions that, when executed by a processing system of a deviceincluding a processor, facilitate performance of operations, theoperations comprising: monitoring network function queues of a pluralityof virtual network functions that are bound to common physicalresources; dumping queueing information for input packets for each ofthe plurality of virtual network functions; identifying from thequeueing information, each of the common physical resources that isunder contention; determining a plurality of network functions in aservice chain that are contenders for the each of the common physicalresources that is under contention; using a diagnosing algorithm todiagnose performance problems based on the determining step; generatingan impact graph for each victim packet identified by the diagnosingalgorithm as a result of a performance problem; and generating a summaryof results of the diagnosing algorithm as a list of rules.
 16. Thenon-transitory, machine-readable medium of claim 15, wherein theperformance problem comprises long tail latency or packet drops.
 17. Thenon-transitory, machine-readable medium of claim 15, wherein using thediagnosing algorithm comprises providing an impact score for each of thecommon physical resources that is under contention.
 18. Thenon-transitory, machine-readable medium of claim 15, wherein identifyingeach of the common physical resources that is under contention comprisesusing resource counters to determine how each of the common physicalresources is under contention.
 19. The non-transitory, machine-readablemedium of claim 18, wherein the resource counters comprise symptomcounters, resource measurement counters and traffic measurementcounters.
 20. The non-transitory, machine-readable medium of claim 15,wherein using the diagnosing algorithm comprises running a scoreback-propagation step for determining which of the plurality of networkfunctions propagate an impact to a victim package.